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Abstract. Let G be a connected and simply connected two-step nilpotent Lie group and T 
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1 Introduction 

Let G be a connected simply connected nilpotent Lie group with Lie algebra q and suppose G 
contains a discrete cocompact subgroup T. Let Rr = Indp (1) be the quasi-regular representation 
of G induced from T. Then Rr is direct sum of irreducible unitary representations each occurring 
with finite multiplicity [3]; we will write 

Rr= J]] m(7r,G,r,l)7r. 

7rG(G:r) 

A basic problem in representation theory is to determine the spectrum (G : T) and the multi- 
plicity function m(7r, G, F, 1). C.C. Moore first studied this problem in [7j. More precisely, we 
have the following theorem. 

Theorem 1. Let G be a simply connected nilpotent Lie group with Lie algebra q and T a lattice 
subgroup of G {i.e., T is a discrete cocompact subgroup of G and log(r) is an additive subgroup 
of g). Let vr be an irreducible unitary representation with coadjoint orbit 0^. Then vr belongs to 
(G : r) if and only if 0^ meets 5p = {/ € Q* , (/,log(r)) C Z} where Q* denotes the dual space 
ofQ- 

Later R. Howe [3| and L. Richardson [12] gave independently the decomposition of Rr for 
an arbitrary compact nilmanifold. In this paper, we pay attention to the question wether the 
multiplicity formula 

m(7r,G,r,l) =#[O^n0f/r] V7re(G:r) 

required in the Abelian context, still holds for non commutative nilpotent Lie groups (we write 
to denote the cardinal number of a set A). In [7] , Moore showed the following inequality 

m(7r,G,r,l) <#[O^n0f/r] Vvre(G':r), (1) 

where T is a lattice subgroup of G, and produced an example for which the inequality ([1]) is 
strict. More precisely, he showed that 

m(7r,G',r,l)2 = #[O^n0j/r] V7rG(G:r) (2) 
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in the case of the 3-dimensional Heisenberg group and T a lattice subgroup. The present paper 
aims to show that every connected, simply connected two-step nilpotent Lie group satisfies 
equation ([2]). We present therefore a counter example for 3-step nilpotent Lie groups. 

2 Rational structures and uniform subgroups 

In this section, we summarize facts concerning rational structures and uniform subgroups in 
a connected, simply connected nilpotent Lie groups. We recommend [2] and [S] as a references. 

2.1 Rational structures 

Let G be a nilpotent, connected and simply connected real Lie group and let g be its Lie 
algebra. We say that g (or G) has a rational structure if there is a Lie algebra over Q such 
that g = 0Q R. It is clear that g has a rational structure if and only if g has an R-basis 
{Xi, . . . , Xn} with rational structure constants. 

Let g have a fixed rational structure given by gq and let f) be an R-subspace of g. Define 
flQ = f) n qq. We say that f) is rational if f) = R-span {Piq}, and that a connected, closed 
subgroup iJ of G is rational if its Lie algebra is rational. The elements of gQ (or Gq = exp(gQ)) 
are called rational elements (or rational points) of g (or G). 

2.2 Uniform subgroups 

A discrete subgroup T is called uniform in G if the quotient space G/T is compact. The 
homogeneous space G/T is called a compact nilmanifold. A proof of the next result can be 
found in Theorem 7 of [5] or in Theorem 2.12 of [llj . 

Theorem 2 (the Malcev rationality criterion). Let G be a simply connected nilpotent Lie group, 
and let g be its Lie algebra. Then G admits a uniform subgroup T if and only if q admits a basis 
{Xi, . . . , Xn} such that 

n 

i^i^^j] = X] '^ijkXk, V 1 < i, J < n, 

k=l 

where the constants Cijk are all rational. {The Cijk are called the structure constants of q relative 
to the basis {Xi, . . . , Xn} ■) 

More precisely, we have, if G has a uniform subgroup F, then g (hence G) has a rational 
structure such that g^ = Q-span {log(r)}. Conversely, if g has a rational structure given by 
some Q-algebra gq C g, then G has a uniform subgroup T such that log(r) C gQ (see [2] and [5]). 
If we endow G with the rational structure induced by a uniform subgroup F and if if is a Lie 
subgroup of G, then H is rational if and only if fl F is a uniform subgroup of H. Note that 
the notion of rational depends on F. 

2.3 Weak and strong Malcev basis 

Let g be a nilpotent Lie algebra and let =^ = {^i, . . . , Xn} be a basis of g. We say that =^ is 
a weak (resp. strong) Malcev basis for g if gj = R-span{Xi, . . . , Xi} is a subalgebras (resp. an 
ideal) of g for each 1 < i < n (see [2]). 

Let F be a uniform subgroup of G. A strong or weak Malcev basis {Xi, . . . , Xn} for g is said 
to be strongly based on F if 

F = exp(ZXi) • • • exp{ZXn). 

Such a basis always exists (see O [21 16] ) . 
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A proof of the next result can be found in Proposition 5.3.2 of [2]. 

Proposition 1. Let T be uniform subgroup in a nilpotent Lie group G, and let Hi ^ i/2 ^ 
••• ^ LLk = G be rational Lie subgroups of G. Let f)i, ■ ■ ■ , f)fc-i) f)fc = 9 be the corresponding 
Lie algebras. Then there exists a weak Malcev basis {Xi, . . . , Xn} for g strongly based on T and 
passing through . . . , If the Hj are all normal, the basis can be chosen to be a strong 

Malcev basis. 

2.4 Lattice subgroups 

Definition 1 Let F be a uniform subgroup of a simply connected nilpotent Lie group G, 

we say that F is a lattice subgroup of G if log(F) is an Abelian subgroup of g. 

In [7], Moore shows that if a simply connected nilpotent Lie group G satisfies the Malcev 
rationality criterion, then G admits a lattice subgroup. 

We close this section with the following proposition (TJ Lemma 3.9]. 

Proposition 2. If T is a lattice subgroup of a simply connected nilpotent Lie group G = exp(g) 
and {Xi, . . . , Xn} is a weak Malcev basis of g strongly based on T , then {Xi, . . . , Xn} is a 'E-basis 
for the additive lattice log(F) in g. 

3 Main result 

We begin with the following definition. 

Definition 2. Let G be a connected, simply connected nilpotent Lie group which satisfies the 
Malcev rationality criterion, and let g be its Lie algebra. 

(1) We say that G satisfies the Moore formula at a lattice subgroup F if we have 

m(7r,G,F,l)2 = #[O^n0j/F], Vvr G (G : F)). 

(2) We say that G satisfies the Moore formula if G satisfies the Moore formula at every lattice 
subgroup F of G. 

Examples. 

(1) Every Abelian Lie group satisfies the Moore formula. 

(2) The 3-dimensional Heisenberg group satisfies the Moore formula (see [71 p. 155]). 

The main result of this paper is the following theorem. 

Theorem 3. Every connected, simply connected two-step nilpotent Lie group satisfies the Moore 
formula. 

Before proving Theorem [31 we must review more of the Corwin-Greenleaf multiplicity for- 
mula. 
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3.1 The Corwin— Greenleaf multiplicity formula 

Using the Poisson summation and Selberg trace formulas, L. Corwin and F.P. Greenleaf [1] gave 
a formula for m(7r, G,T, 1) that depended only on the coadjoint orbit in g* corresponding to vr 
via Kirillov theory. We state their formula for lattice subgroups. Let F be a lattice subgroup of 
a connected, simply connected nilpotent Lie group G = exp(0). Let 

flr = OG0*: (Z,log(r)) CZ}. 
Let TT/ be an irreducible unitary representation of G with coadjoint orbit C g* such that 
^ i^}- According to Theorem [U we have m{7Ti,G,T, 1) > if and only if 0^^ ngf / 0, so 
we will suppose this intersection is nonempty. The set figp is F-invariant. For such F-orbit 
ft C 0^ CiQy one can associate a number c(0) as follows: let / € r2 and g(/) = kei{Bf), where 
Bf is the skew-symmetric bilinear form on q given by 

Bf{X,Y) = {f,[X,Y]), X,Y€0. 

Since (/, log(F)) C Z then g(/) is a rational subalgebra. There exists a weak Malcev basis 
{Xi, . . . , Xn} of g strongly based on F and passing through g(/) (see [21 Proposition 5.3.2]). We 
write g(/) = IR-span{Xi, . . . ,Xs}. Let 

Af = Mat((/, [Xi,Xj]) : s<i,j<n). (3) 

Then det{Af) is independent of the basis satisfying the above conditions and depends only on 
the F-orbit $7. Set 

c{n) = {det{Af))-K 

Then c(r2) is a positive rational number and the multiplicity formula of Corwin-Greenleaf is 

1, if 0(0 = 9, 

c(r2), otherwise. (4) 

ne[o^ ng*/r] 



m(7r;,G,F,l 
For details see [1]. 



Proof of Theorem [3l Let I G O^Pigp. The result is obvious if q{1) = g. Next, we suppose 
that g(/) / g. Since G is two-step nilpotent Lie group then g(/) is an ideal of g, and hence 
we have g(/) = g(/) for every / e 0^ and 0^ = Z + q{1)-^ (see [21 Theorem 3.2.3]). On the 
other hand, as I belongs to gj. then g(/) is rational. By Proposition 5.3.2 of [2] there exists 
a Jordan-Holder basis =^ = {Xi, . . . of g strongly based on F and passing through g(^). 

Set g(/) = E -span {Xi, . . . , ^s}- 

Then, for every Q € [0^ ng^/F] and for every / G ^2, we have 
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c{n) = det(^/)-2 = det{Ai)-2 = c(F • /), 
since /|[g,g] = ^|[g,g]- It follows from ^ that 

m(7r,G,F,l) = #[0^ngf/F] c(F.O. (5) 
Next, we calculate #[0^ ngf,/F]. Let {h, . . . , t„) G and / G 0^ Rgf . We have 

n / n \ 

(exp(-tiXi) ■ • • eM-tnXn)) • / = / + Yl E [Xj,Xi]) X* 

i=s+l \j=s+l J 

n I n \ 

= /+ E E hUx,.x,\)\xi, 

i=s+l \i=s+l / 
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since /|[g,g] = l\[g,g]- It follows that 

n 

r-/ = /+ Yl ^^i' 

j=s+l 

where 

n 

ej= {l,[Xj,Xi])X:, ys<j<n. 

Let 

n 

4 = qG ng* - / = ZX* and = ^e,-- 

s<i<n j=s+l 

Since g(/)nR-span {Xg+i, . . . , Xn} = {0}, then the vectors e^+i, • • • , are linearly independent. 
Therefore, fi.o is a sublattice of fi.. It is well known that there exist £s+i, ■ ■ ■ ,£n a linearly 
independent vectors of g* and ds+i, . . . , (i„ € IN* such that 

Jt = Ze, and = di'Ze^. 

s<i<n s<i<n 

Consequently, we have 

Let [e^+i, ...,£„] be the matrix with column vectors £s+i, ■ ■ ■ ,£n expressed in the basis (X*^-^, 
...,X:). From 

i = ZX* = Ze„ 

s<i<n s<i<n 

we deduce that 

[Es+i, . . . ,en] G GL(?i - s,Z). 

On the other hand, let [cg+i, . . . , e„] (resp. [d s+i£ s+i, ■■■ ,dn£n]) be the matrix with column 
vectors e^+i, . . . , (resp. ds+i£s+i, ■ ■ ■ , dn.^n) expressed in the basis {X*_^_-^^, . . . , X*). Since 

Jto = ^ ^ej = d,Zei, 

j=s+l s<i<n 

then there exists T G GL(n — s, Z) such that 

[Cs+l, • • • , e„] = [ds-i-lEs+l, . . . , dnSn]T. 

The latter condition can be written 

= [£s+i, • • • , en]diag[4+i, . . . , dn]T. 
Form this it follows that 

det(A) = ds+i ■■■dn- 
Consequently 

#[o^n0j/r] = det(^O- (6) 

Substituting the last expression ([6]) into ([5]), we obtain 

m(7r,G,r,i)2 = #[o^n0f/r]. 

This completes the proof. ■ 
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As a consequence of the above result, we obtain the following result. 

Corollary 1. Let G he a connected, simply connected two-step nilpotent Lie group, let g be the 
Lie algebra of G, and let T be a lattice subgroup of G. Let I G g* such that the representation vr/ 
appears in the decomposition of Rr • Let Ai as in The multiplicity of iri is 



Remark 1. Note that in [inj, H. Pesce obtained the above result more generally when F is a 
uniform subgroup of G. 

4 Three-step example 

In this section, we give an example of three-step nilpotent Lie group that does not satisfy the 
Moore formula. Consider the 4-dimensional three-step nilpotent Lie algebra 

= R-span {Xi, . . . , X4} 

with Lie brackets given by 



and the non-defined brackets being equal to zero or obtained by antisymmetry. Let G be the 
simply connected Lie group with Lie algebra g. The group G is called the generic filiform 
nilpotent Lie group of dimension four. Let F be the lattice subgroup of G defined by 

F = exp(ZXi)exp(ZX2)exp(ZX3)exp(6ZX4) = exp(ZXi © ZX2 © © 6ZX4). 

Let I = X^. It is clear that the ideal m = ]R-span{Xi, . . . ,^3} is a rational polarization at I. 
On the other hand, we have (Z,mnlog(F)) C Consequently, the representation tti occurs 
in Rr (see [Kill]). Now, we have to calculate #[0^^ ng^/F]. 
Following [2] or [8], the coadjoint orbit of / has the form 




[X4, Xi] — Xi-i 



i = 2,3 




On the other hand, it is easy to verify that 




Therefore 




Let 



•0 



XI + toX^ + 1^3* + ^^4* G 0^^ ng*r 



and 



7 = exp(rX2)exp(sX3)exp(6tX4) G F. 
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We calculate 



Ad*(7)/to,so = XI + {to - 6t)X* + + (^ + sto + r- 6st) X*. 
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Then (see [S]) 



Ad*(r)^,,„ = |x* + (to + 6t)x* + (^o + 6t)^ ^, + (f + ^) ^4 : 5, t e z| 

= {/to+6t,so+6s : s,t e Z} . 
From this we deduce that #[0^^ ngf/F] = 3 • 6 = 18, and hence 

m(7r,,G,r,i)V#[o^,n0f/r]. 

Therefore, the group G does not satisfy the Moore formula at F. 
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